Robust Markov Decision Processes for Medical Treatment Decisions

Medical treatment decisions involve complex tradeoffs between the risks and benefits of various treatment options. The diversity of treatment options that patients can choose over time, and uncertainties in future health outcomes result in a difficult sequential decision making problem. Markov decision processes (MDPs) are commonly used to study medical treatment decisions; however, optimal policies obtained by solving MDPs may be affected by the uncertain nature of the model parameter estimates. In this article, we present a robust Markov decision process treatment model (RMDP-TM) with a controllable uncertainty set formulation for the transition probability matrices (TPMs) of the underlying Markov chain. We show that the RMDP-TM can overcome the common problem of over-conservativeness of the worst-case optimal policy obtained from the RMDP model with fixed uncertainty set formulations for TPMs. We present theoretical analysis to establish computationally efficient methods to solve the RMDP-TM and present its application to a medical treatment decision problem of optimizing the sequence and the start time to initiate medications for glycemic control for patients with type 2 diabetes.